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PREFACE 



The mission o£ the Intelligent Training Branch of the 
Technical Training Research Division o£ the Human Resources 
Directorate o£ the Armstrong Laboratory (AL/HRTI) is to design, 
develop, and evaluate the application o£ artificial intelligence 
(AI) technologies to computer-assisted training systems. The 
current e££ort was undertaken as part o£ HRTI's research on 
intelligent tutoring systems (ITS) and ITS development tools. 
The work was accomplished under workunit 1121-10-49, Artificial 
Intelligence in Training. The proposal for this research was 
solicited using a Broad Agency Announcement. 
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Summary 



The objective of die current research program was to develop a prototype knowledge acquisition 
tool to assist in Hht investigation of knowledge acquisition issues for instructional systems. Our 
approach entailed identifying the types of knowledge to be acquired, evaluating metiiods of 
acquiring such knowledge, speci^g appropriate interaction metaphors for the direct acquisition 
of knowledge from multiple human sources, and iteratively developing an Artificial Intelligence in 
Training (ATT) prototype that operationalized tiie findings of the first three tasks. Hie current 
research effort produced a number of knowledge acquisition tool requirements for training 
troubleshooting activities in complex systems. This domain requires die integration of at least three 
different types of knowledge: (1) knowledge about systems structure, function and tiieir normal 
and abnormal behavior (how it works), (2) knowledge about interacting with these systems to 
perform a task (how to use it), and (3) knowledge about how to teach stiidents to perform a task 
(how to teach it). In addition, tiie following principles were specified for a knowledge acquisition 
environment designed to acquire these types of knowledge: 

• Knowledge acquisition should take place in a sunulated environment in which users 
interact directiy with a representation of their domain that emulates their real- world 
interactions and pronaotes the solicitation of natural and instinctive responses 
through a direct manipulaticm interface. 

• The toiowledge acquisition tool should employ an action-based solicitation method 
in which all user actions are recorded in sequence and in "pseudo-real-time." 

• The knowledf ^ acquisition tool should support knowledge integration by providing 
users witii multiple, animated views of knowledge tiiat are completely introspective 
and manipulatable. 

• The knowledge acquisition tool should support knowledge construction from 
observations of user acticms. 

The following areas for additional research were recommended: Acquisition of instructional 
heuristics, dynamic constraction of justification knowledge, use of higher level problem 
representations such as die goal-action hierarchy, and most importandy, die accuracy of die 
expertise cq)tured using diese mediods. By demonstrating new approaches and integrating 
estabUshed approaches to knowledge acquisition, we hoped to stimulate additional research on 
apiroaches to die direct acquisition of multiple knowledge types fiom multiple knowledge sources 
for intelligent instructional environments. 
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Section 1 
Introduction 



Methods for faciUtating inteUigent tutoring systems (ITS) development aie key concerns in both 
the Air Force (Regian, 1989) and industry (Bloom, BuUemer, and Cochian, 1989). Of the many 
technical challenges facing the developers of ITSs, the most chaUenging and intransigent revolve 
around the issue of knowledge acquisition. Traditional knowledge acquisition problems include the 
knowledge acquisition botdencck, the knowledge filtration problem, cognitive biases, and the 
difficulty in ''unpacking** compiled expertise. Compoundmg these traditional knowledge 
acquisition problems, knowledge acquisition for instructional systems introduces additional 
problems, including the need to acquire multiple knowledge types from multiple sources (i.e., 
various types of domain and instructional knowledge), the need for the knowledge input by one 
expen to be completely understood by another expert, necessitating the acquisition of "deep" 
knowledge that can be both inspected and manipulated (Le., actions onrf justifications the 
acquisition of which previously required knowledge engineers to independendy employ a variety 
of direct and mdirect knowledge acquisition methods), and the fact that the acquisition of • 
instructional knowledge and dwnain knowledge is interrelated. 

Employing the traditional knowledge acquisition scenario of having knowledge engineers acquire 
knowledge from domain experts and instructional developers separately for an ITS is untenable for 
a variety of reasons. First, there is the long and costly development cycle produced by this 
scenano. Second, because knowledge engineers use a variety of widely differing direct and indirect 
knowledge acquisition methods, tiie knowledge acquired can often be infiiscd with inaccuracies 
making it a "best guess," rather than a true representation of expertise. Third, because tiicrc are ' 
many domams represented, the knowledge input by an expert of any one domain must be 
accessible to and understood by an expert of any other domain. 

iTie objective of the current research program was to develop a prototype knowledge acquisition 
tool tiiat can assist in the investigation of knowledge acquisition issues for instructional systems by 
demonstratmg new approaches or integrating established approaches to knowledge acquisition tiiat 
will help illuminate research questions regarding the direct acquisition of multiple knowledge types 
from multiple knowledge sources. The approach employed to accompUsh this objective conmrised 
fow major tasks. The first task involved identifying the types of knowledge to be acquired for an 
rrS. TTie second task involved evaluating methods of acquiring such knowledge, including tfie 
exploration and evaluation of new alternatives if necessary. The tiiird task involved defining 
appropriate metaphors of interaction to faciUtate tiic direct acquisition of knowledge from human 
sources witfiout needing die help of knowledge engineering or AI programming specialists. The 
fourth and final task involved itcratively developing a prototype that operationalizes the above 
findings. 

To summarize briefly, die solution being proposed in die current project has several aspects to it 
First, domain experts and instructional developers must interact diiwtly with die knowledge 
acquisition tool. This should alleviate some of the traditional knowledge acquisition problems 
associated wi A having a knowledge engineer in die loop, such as die knowledge acquisition 



I?) 



botdcncck and knowledge filtration problems. Second, the knowledge acquisition metfiod(s) 
employed must be specifically designed to promote natural and instinctive responses by both 
domain and instructional experts. To demonstrate this aspect, we arc proposing the following: 

• Provide experts and instractors with an environment that closely emulates their real- 
woiid interactions with thcu: domaia* 

• Ensure that the knowledge input by any one expert can be inspected and manipulated 
by any other. This is an instantiaticm of the notion of a "glass box architecture'* 
(Anderson, 1988; Wengcr, 1989), in which we enable multiple views and 
representations of the acquired knowledge that can be manipulated 

• Employ a knowledge acquisition method tiiat is able to obtain information both 
about what experts do and why they are dcrnig it (Gruber, 1991) in a manner tfiat 
does not distract them from their primary task. 

Third, the tool must be capable of both supprating and integrating multiple types of input from 
multiple experts to ensure that the knowledge acquired is as ftce from individual cognitive biases 
as possible. 

This report is organized as follows: Section 2 contains a descripticm of critical issues in knowledge 
acquisiticm, including those introduced by expanding knowledge acquisition to the domain of 
intelligent tutoring systems. Section 3 describes ihc knowledge acquisition approach proposed to 
address these issues, including discussions of tiie prototype's knowledge requirements and 
concepts underlying the knowledge acquisition capabilities implemented in the prototype. Section 4 
contains scenarios dq>icting how sample knowledge acquisition sessions might be conducted 
using tiie AI in Training (ATT) prototype. Section 5 contains conclusions reached ficom the current 
effort Section 6 contains recommendations and a discussion of future research directions and 
issues to be addressed in the area of knowledge acquisition for ITSs. 
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Section 2 

Critical Issues in Knowledge Acquisition 



The problem of knowledge acquisition is one of the biggest issues affecting the development and 
delivery of artificial intelligence (AI) systems such as expert systems and intelligent tutoring 
systems. Early AI systems were plagued by long and costly development cycles, driven primarUy 
by the difficulty of knowledge acquisition. The most successful attempts to address these 
problems have involved developing systems in which the knowledge engineers and AI 
programmers have been removed fixra the knowledge enguieeiing loop entirely. 

So far, however, such automated knowledge acquisition has been possible only for specific 
domains. While the resulting domain-specific shells are very effective in providing solutions to 
tasks for which they were designed, they are not easUy generalized to new areas. Furthermore, it 
has been weU estabUshcd that expertise is made up of several different kinds of knowledge (Berry, 
1987; McGraw & Scale, 1988). When expanding a system's scope ftom expert advisory to 
instruction, failure to capture and represent different knowledge types, which could come from 
multiple sources (i.e., domain expert, cognitive scientist, teaching expert) could result in a mtor that 
IS both limited in scope and lacks the flexibiUty of a human tutor (Woolf and Cunningham, 1987). 

Some of the critical issues in knowledge acquisition revolve around the problems of tiie 
knowledge acquisition botdencck and knowledge filtration, the nature of compiled expertise, and 
cognitive bias. Additional issues present tiiemselves when designing a system for instruction as 
opposed to a system for performance. These issues are discussed further in tiie subsections below. 



2.1 Knowledge Acquisition Bottleneck 

At present, knowledge acquisition for AI systems, in general, is very inefficient. In the early days 
of AI systems development, a typical development scenario would have a knowledge engineer 
interviewing a subject matter expert to acquire the knowledge to be incorporated into the system. 
The knowledge engineer would in turn interact with an AI programmer to convert the acquired 
knowledge into an identified AI representation written in a general-purpose AI language, such as 
OPS-5, LISP, or Prolog. This process of characterizing the knowledge of experts is the significant 
bottieneck in intelligent systems development The obvious drawback to this methodology was tiiat 
the high-level data structures and reasoning schemes that represented the domain knowledge had to 
be built fiom scratch for each new appUcation. The technique amplifies the inefficiency of software 
development, requiring knowledge engineers to write one kind of program and programmers to 
conven it to another. 



2.2 Knowledge Filtration 

In addition to die bottieneck problem, tiiere is also the filtration problem, which refere to uie 
multiple transformations pcifoimed on knowledge as it passes ftom expen to knowledge engineer 
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to programmer to software. Inaccuracies in the system^s encoded knowledge can result from the 
filtering or processing of that knowledge by two nonexperts prior to its entry in the system 
(Cochran, Bloom, and BuUemer, 1990X 

23 Compiled Expertise 

The solution to these problems is obvious yet difficult to accon^lish: eliminate the need for 
knowledge engineers and programmers developing authoring environments easy enough for 
domain experts to use unassisted The essoitial difficulty of this strategy is that expertise is not so 
easily acquired. One of the fundamental difficulties in most knowledge acquisiticMi methods is that 
the human thinking process we wish to understand and model is not available to direct 
observation. This difficulty is furtiier compounded by the fact that expertise is typically not 
reportable, due to the compilation of knowledge that results from extensrve practice in a domain of 
problem-solving activity (Anderson, Grceno, Kline and Neves, 1981). Because experts "see" the 
solutions to complex problems more often than they deduce them, their expertise is said to be 
"compiled,"' that is, the individual elements of expertise are not available. Thus, Imowledge 
engineers tend to use a variety of different knowledge acquisiticm techniques and methods 
developed to try to gain access to the experts* reasoning processes. Olson and Rueter (1987) 
reviewed a number of these methods of knowledge acquisition, classifying them into direct and 
indirect methods. Table 1 lists a number of different knowled^ acquisition methods: 



Table 2. Methods of Knowledge AcquiMon 



Direct Methods 


Indirect Methods 


Interviews 
Questionnaires 

Observation of Task Performance 
Protocol Analyses 
Interruption Analysis 
Drawing Closed Ounces 


Multidimensional Scaling 
Hierarchical Clustering 
Weighted Networks 
Ordered Trees 
Repertory Grid Analysis 
Device Models 



All the direct methods for acquiring knowledge listed in Table 1 can elicit a wide range of 
knowledge types. However, tiieir drawback is that until now, they have been techniques employed 
by knowledge engineers, not techniques in4>lemented in a computer-based knowledge acquisition 
tool usable directiy by domain experts. On die other hand, even though all the indirect metiiods 
lend themselves to being implemented in knowledge acquisition tools, and in many cases, 
individually, they have been, they illuminate only particular types of domain expertise making 
them unsuitable for the task of acquiring the multiple types of knowledge required by an ITS. In 
addition, each of these methods was developed as a compromise iac dealing with the problems 
elaborated on previously. The result of applying any one of these methods is the acquisiticHi of 
knowledge that can be biased both by the method itself and by the representation en^>loyed by that 
method. 
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For example, the repertory grid analyas approach to acquiring expert knowledge begins by 
specifying a taxonomy of domain concepts and their interrelationships, such as listing problem 
features, solution states, and die relationship between problem features and solution states (Boose, 
Biadshaw, Kitto, and Shema, 1989X Procedural rules are then derived that reflect the underlying 
functional model of the domain. Although this approach allows experts to interact direcdy with the 
knowledge acquisition tool, entering their knowledge using thdr own vocabulary, heuristic 
knowledge is not ciq)tured directly using this approach. Drclarative knowledge of the domain 
concepts is captured and manq>ulated to produce if*dien rules that are used to simulate or model 
task performance. The resulting procedural knowledge is derived independent of any specific task 
situation, often failing to capture an e?q>ert's heuristic knowledge. 

What is needed is a computer-based knowledge acquisition method that ccMnbines the best of both 
the direct and indirect methods. First, it is important to be able to acccnnplish on a c<Mnputcr what 
knowledge engineers accon5>lish using direct acquisition methods. It is widely acknowledged that 
the best way to discovo" how experts make a judgment, diagnosis or design decision is to watch 
them work on a real problem (Olson and Rueter, 1987). Therefore, what would be desirable would 
be to inclement a knowledge acquisitiai method that automatically records all Ae experts* actions 
(i.e., what they did, when they did it, and tfie order of thdr actions) while tiiey are engaged in a 
real-world-like problem solving activity on the computer. In addition, this computer-based 
knowledge acquisition tool needs to be able to accon^lish what knowledge engineers accomplish 
using indirea methods: the unpacking of e)q>ertise in the form of explanations and justifications 
for the various actions an e?q)ert takes and decisions the expert makes. 

According to Gmber (1991), justificaticms are declarative specificaticms of what the situation is, 
what the possible choices or actions are, and a set of reasons, defined as specifications of relevant 
features of the situaticm or choice, for a specific choice being appropriate. To determine why a 
choice is. appropriate in a given situation it is necessary to solicit a set of relevant features of both 
the situation and the choice from the expert Solicitation of these features should take place within 
the context of a specific exan^le in a running system in which the features can be computed. 
Within the domain of diagnosis, whether it be naedical or complex-system faults, simations can be 
defined as diagnostic steps or hypotheses and choices can be defined as the diagnostic tests the 
diagnostician performs to rule out or confirm those hypotheses. 

2 A Cognitive Bias 

An additional problem that has always affected expert systems, one that unfortunately becomes 
amplified by allowing domain experts to develqp expert systems directly, is tiiat the expert system 
is only as good as tiie expert (Qeaves, 1987; Meyers and Booker, 1989). Often, cognitive biases, 
such as inaccurate or incomplete domain representations, are accidentally built into a kiK>wledge 
base. Cognitive bias can result ftom either limitations in an expert's knowledge or expertise, ot 
ft'om the expert having to generate procedures fiom memory during knowledge acquisition. 
Cognitive limitations in recalling or simulating the tasks could also produce gi^s or inaccuracies in 
tiic knowledge acquired While tiiis may not greatiy affea the utility of advisory e)q)ert systems (if 
the system is as good as a carefully selected expert in solving problems in a particular domain, any 
cognitive bias in the approach to those solutions is an advantage), the interaction of cognitive 
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biases can lead to significant problems when the knowledge is used for more than one purpose 
which IS the case for training systems. ' 

One solution to the cognitive bias problem requires system developers to provide more 
smiultaneous assistance to experts. For example, knowledge engineers are skilled in detecting 
some of the symptoms cf bias and in questioning experts to ascertain the gaps in their knowledge 
fa an automated knowledge acquisition tool, much of this assistance could instead be provided by' 
havmg the expert(s) work in an environment that closely emulates their leal-world domain 
interactions. By engaging experts in interactions with an emulated environment that responds to 
their actions as their actual domain would, and by constraining those interactions to environment- 
legal activities only, one provides the context necessary to avoid or detea biases as aresult of the 
experts cognitive limitations. 

Another solution to the cognitive bias problem involves the use of multiple experts for a single 
type of domam knowledge. The use of multiple experts in this sense reduces the likeUhood that 
Idiosyncratic and/or maccurate knowledge will be encoded in the knowledge base. In an automated 
knowledge acquisition tool, this could be accomplished by developing an approach for acquiring 
expemse distnbuted across individuals, that is, it would allow experts to input their knowledee 
separately, combined with faciUdes to review, evaluate, and integrate the acquired distributed 



2.5 Knowledge Requirements 

A knowledge base for use in a training system will need to be able to represent all the expertise of 
a specific domam as well as the various types of knowledge used in the expression of that 
^^"^^■^ ^"""^ °^ knowledge that could be acquired for any domain and the likelihood 
that the different types of knowledge will come fiom different sources, a different type of multiple 
expert problem is created. As such, additional issues that must be considered in designing a 
knowledge acquisition tool for instructional systems arc identifying the types of knowledge to be 
acquired and the relationships between those types of knowledge. 

The objectives of a system designed for instruction are different from the objectives of a system 
designed for task peiformance. Consequently, the knowledge acquisition requirements will be 
different as well. For example, Qancey (1984) reported that an attempt to use MYCIN as a basis 
for teaching expert diagnostic skill proved difficult because specific types of information were 
either absent or mipUcitly represented in the systems production rules. Clancey reported that using 
an expert system to teach requires a shift in orientation (objective) ftom simply trying to generate 
gcxxl task solutions to simulating in some degree of detail the reasoning processes itself. Types of 
infOTmation required depend on the instructional objectives and methods. An ITS requires more 
explicit, psychologically valid models of task performance. 

A knowledge acquisition tool for instructional systems will need to be able to acquire a number of 
different types of inteirelated knowledge. To teach problem solving in complex systems, it wiU be 
necessary to acquire knowledge about the system itself, about the types of problems one can 
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encounter in that system, about the ways experts solve those problems, and about the ways 
instructors teach that knowledge. 

To begin with, to emulate the cxperVs real-war!d interactions in a dwnain, the knowledge 
acquisition tool will need to possess and use an abundance of knowledge about the structure, 
functiwi, and behavior of the various devices, subsystems and systems of the domain, that is, 
device knowledge. When the expert takes an action on some object in the knowledge acquisition 
tool's domain representation, the represented system's reaction to that actiwi shouU emulate tiie 
reaction expected in the actual systeia Next, tiie knowledge acquisition tool will need to acquire 
knowledge about problems. The basis for teaching problem solving knowledge, or acquiring 
problem solving expertise, is direct interacticm with domain problems. One approach to 
accomplishing tiiis is to enable experts to build libraries of real domain problems, reflecting both 
common and uncommon occurrences. This will involve the initialization of input and control 
values and parameters as well as the introducti<m of faults (and their consequences) into the system 
being represented. This acquired problem knowledge can then be used to initialize tiie tool's 
representation of tiie device knowledge to create scenarios around which to conduct the "real-time" 
acquisition of problem solving knowledge. This will involve recording tiie actions die expert takes, 
as well as die represented system's behaviors in response to tiiose actions, and constructing 
justifications for tiiose actions to give tiic problem solutions explanatory power. Hnally, tiie 
knowledge acquisition tool wiQ need to support ti^e instructor's use of all tiie previously acquired 
knowledge in instructional interactions witii "students." This will involve enabling die instructor to 
review and comprehend die problems being solved and die expert approaches to solving tiiose 
problems so tiiat tiiey can evaluate tiie student's performance and deptii of knowledge. In addition, 
the tool rieeds to support metiiods for recording die instruaor's interventions and construct 
justifications for those interventions firom the recorded acticHis. 

In die section tiiat follows we will describe die ATT system at a conceptual level, including a 
discussion of die system's knowledge requirements and descriptions of die various features of die 
knowledge acquisition environment developed in response to die issues elabc^ted on previously. 
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Section 3 
Current Approach: AIT 



3.1 AIT Knowledge Requirements 

The ATT demonstrates acquisition of knowledge used to support training of troubleshooting 
activities in a process control domain. Support for the training and performance of troubleshooting 
activities in the process control domain is provided by three basic types of knowledge. These 
include: 

• Knowledge about systems and their normal and abnormal processes and behavior 
(how it works), 

• Knowledge about interacting with these systems to perform a job or task (how to 
use it), 

• Knowledge about how to teach students to perform a job or task (how to teach it) 
(adapted from Kieras, 1988). 

3.1.1 ''HowM-Works'' KnowUdge 

A basic knowledge requirement for a simulation-based ITS is device knowledge. Device 
knowledge characterizes the structure, function and behavior of a process control system. System 
designers are typical sources of expertise regarding device knowledge, particularly with respect to 
normal structure, function and behavior. Field technicians and engineers are typical sources of 
expertise for knowledge about a device's abnormal structure, fimctim and behavior. System 
designers often have less experience with abnormal behavior. 

Table 2 contains a list of comnwn representations of device knowledge. These representations 
contain knowledge about the physical layout of major systems, subsystems, components and their 
interconnections; the flow of data between system components; and normal behavior of the system 
in terms of conditional state transiticms or cause-effect relaticxiships. 

There are a number of research programs cunentiy investigating methods of acquiring and 
building device model representations (e.g., IMTS, ICATT) as well as commercially available 
tools for building object-based system simulations. To avoid duplicating these efforts, it was 
decided that the current effort would not address this issue, assuming instead that <xie of the 
methods existing or under investigation could be incorporated into the ATT system to support the 
acquisition of device knowledge. Instead, it was decided to represent device knowledge to the 
extent necessary to support the acquisition of the other knowledge types (see Section 3) and to do 
so in an economical and efficient manner. As such, it was decided to represent device knowledge 
in ATT using a qualitative behaviofal model of a process control subsystem at the level of 
replaceable components. A qualitative behavioral model enables the system to simulate the 
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Tabu 2. Real-World RepreuMtadoiu af Device Kmowteige 



Dtvica Knowltdy Riir—Mrtatlont 

Component operating characteristic (s^em specifications) 
Process control descriptions (control schematics) 
General physical principles (physics) 
System operation descriptions (manuals) 
Functional block diagrams 
Data flow diagrams 

Component-level schematic diagram (usually at subsystem level) 
PIP drawings 



behavior of the subsystem. The behavioral model relates quaUtative changes in conroonent 
Fopcmes to quaUtative changes in the consent's output The specification of the causal relation 
between each component's input and output enables the quaUtative simulation of the subsystem's 
normal behavior. The interface presents a component-level schematic diagram with access to the 
mput and output of the subsystem and properties of the individual components, such as percent 
opening of a valve. 

Because the task domain of the ATT is troubleshooting, the behavioral model must be able to 
simulate abnonnal behavior as weU as normal behavior. Knowing about how a device can fail 
mvolves knowing what conqxMients can fail and what the impact would be on the behavior of the 
system. Thus, the device knowledge of ATT represents failure states of each component and their 
impact on the con^nent's output 

3.12 "How-to-Use-it" Knowkdge 

"How-to-use-it" knowledge is the knowledge a user has about interacting with a system to 
pcrfonn a job or task. The scope of the "how-to-usc-it" knowledge in the ATT demonstration 
prototype was Umited to troubleshooting system feults in a simpUficd, prototypical process control 
appbcation. Troubleshooting can be considered a diagnostic problem solving task, requiring die 
Identification of changes in nomial system behavior (deviation fiom expected operation) and 
locaUzation of die causes of abnormal behavior (Rasmussen, 1981). This means tiiat in addition to 
an understanding of the appUcation's device knowledge tfiat enables experts to identify instances of 
normal and abnormal system behavior, troubleshooters need to possess two types of "how-to-use- 
it knowledge: problem knowledge and problem solving knowledge. 

Problem knowledge can be defined as knowledge of the specific types of problems, symptoms 
and faults for a system as weU as knowledge of die mappings ftom symptoms to faults and 
problems, tiiat is, knowing what components can fail and how Aey can faU. Sources of tiiis type of 
knowledge are typically field technicians (operators, troubleshooters) and in rare cases, system 
designers. ' 
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Problem knowledge in ATT includes an initial symptom description, a fault state (property) and a 
value for tiiat fault state. Classes of faults have been enumerated in ATT's device knowledge. 
Problem specification involves describing the initial symptoms, selecting components and either 
initializing input values or choosing a fault fmn a list of potential faults and assigning a value to 
tiie fault Circumstantial infonnation and fault probability knowledge can be included using 
explanations attached to specific problem setup actions. 

Problem solving knowledge can be characterized as knowledge about problem solving strategies 
used to generate problem solving goals and knowledge about procedures for manipulating and 
testing the system. The diagnostic probl«n solving task can be described as follows: It begins witfi 
the identification of some initial problem state (initial symptans) that indicates that the system is 
behaving abnormally. Next come tfie generation of a set of possible faults (hypotheses) tiiat could 
have produced tiie initial symptoms. Finally, there is the pcrfonnance of infwmation gatficring 
activities that allow the problem solver to confirm or rule out hypotheses in the hypotheses set 

Diagnostic problem solving expertise is often tiiought to consist of heuristics (rules of thumb) for 
generating possible problem hypotiieses and for evaluating tiie hypotiiesis given available 
information (Clancey, 1985) as well as the strategic knowledge tfiat guides problem solvers in 
deciding what action to perform next (Gruber, 1991). Problem solving expertise is acquired with 
years of experience in performing a task in a given domain. Heuristics are often represented as 
associations between problem symptoms and fault states. 

Gruber (1991) has proposed tiiat strategic knowledge can be represented in terms of justifications 
for specific problem solving actions. Justifications are specified in terms of the relevant features of 
the current problem situation that explain why an action taken is the appropriate action. Relevant 
features of a current situation include problem solving goals, current hypotheses, and knowledge 
about the outcome of tests that provide evidence for or against current hypotheses. 

AIT represents problem solving knowledge in terms of sequences of actions used to localize 
problem causes and the generation and elimination of problem hypotheses from a hypothesis 
space. Problem solving actions include viewing available component property information and 
conducting tests to obtain additional conq)onent state information. 

In AFT, strategic knowledge is represented in AIT in terms of task objectives and justifications for 
performing problem solving actions. Justifications are acquired in two ways. First, ATT 
dynamically generates justifications for each user action in terms of its impact on tiie problem 
hypothesis q>ace. Each action can add and/or rule out hypotheses from the hypotiiesis space. 
Second, a user may attach an explanatory statenoent to any problem solving action diiectiy. In 
addition, following conq)letion of their problem solving activity, experts arc instructed to specify a 
goal-action hierarchy tiiat shows tfie relation between task objectives and tiie activities perfOTmed to 
achieve tiiose objectives. The goal-action hierarchy contains goal and subgoal states that are linked 
to each of the knowledge acquisition actions. 

AIT supports acquisition of problem solving knowledge by observing and recording problem 
solving actions and by soliciting and updating a problem hypotiiesis space. ATT records user 
actions in accessing component information and pron^ts users to add and/or remove hypotheses 
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following each infonnation acquisitiwi action. For each problem solutitm, users specify a goal- 
action hierarchy. Users enter goal and subgoal desoipticms with links to problem solving actions 
showing the relation between task objectives and the activities perfonned to achieve those 
objectives. 

3,13 "HoW'tO'Teach-it*' Knowledge 

Muiiay and Woolf (1991) have defined Instructional knowledge as "the knowledge related to 
teaching and learning a given domain," including in their definition only the types of knowledge 
that we consi jo- to be instructional plan infonnation: knowledge about the structuring of a domain 
(e.g., the selection and sequencing of lessons and topics) and the identification of parameters or 
"bugs" to monitor for smdent evaluation purposes. 

Thwe has been considerable research undertaken to develop tools and methods to facilitate the 
acquisition of instruction plan informatiCHi (Bonar, Cunningham, and Schultz, 1986; Macmillan, 
Emme, and Berkowitz, 1988; Merrill, 1989; Murray and Woolf, 1991; P-isseU, Moran, and 
Jordan, 1988). Murray and Woolf have developed a prototype knowledge acquisition system 
called KAFTTS (Knowledge Acquisition Framework for Intelligent Tutoring Systems), which was 
designed to acquire and represent knowledge fiom instractional experts. KAFTTS incorporates 
instructional design paradigms and facilitates the rapid creation and manipulation of multiple 
tutoring strategies. Merrill (1989) has developed an instructional system design expert system for 
instructional designers that guides instructional design decisions so that resulting products can 
more adequately implement what is known about learning and instructicmal design. Russell et al. 
(1988) have developed the instructicmal design environment (IDE), a sophisticated, integrated 
computer-based environment for instructional design. IDE facilitates the instructiOTal design 
process and the creation of instructional materials by providing a hypermedia-based, flexible 
representation workbench. In IDE, all the information used for designing and developing a course 
can be represented and manipulated. IDE sin^lifies the course development task and enables the 
construction of more consistent instruction by providing tools and structures that automate many 
of the routine tasks. 

Our research indicates that there is an additional categoiy of instmctional knowledge— knowledge 
about how instmctors dynamically and ^ntaneously interact with a studait— tiiat has either been 
ignored or else dismissed as being irrelevant or too difficult to obtain (Murray and Woolf, 1991). 
We call this type of instmctional knowledge instmctional heuristics, and it is the premise of the 
current research effort that acquiring this type of instructional kr wledge fw an TTS is critical, 
particularly within the context of teaching problem solving in complex systems, a significant 
problem for industry and the military. 

Human instructors respond to student performance in a number of different ways, for example, 
providing advice or demonstrating how to perform a task. Instructional heuristics specify the 
instructional situations for different types of instructional responses. Situations can be defined in 
terms of features of student troubleshooting actions such as type of infamatiaj accessed, type of 
diagnostic test performed, or relation of student strategies to expert strategies. The types of 
instructional responses can vary from making certain types of advice available to presenting an 
expert solution to tiie problem. Table 3 lists a number of types of intervention derived fiom the 
research of Bunon (1988) and Wenger (1989). 
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Table 3. Types of Assimnee 



Type of 
IntervMtlcn 


Description 


Prototypical Instructor 
Action 


Examples 


Assistance 


The instructor takes over parts of the 
problem-solving task, freeing the 
student to concentrate on the 
remaining parts. TMs type of 
interventton is usually applied to 
nonessential parts of the task that 
may be bk>cking the student from 
progressing and grasping the larger 
structural properties of the domain. 


Instructor provkJes 
student with 'answer 
to that part of the 
problem solving task. 


Do-for, remind 


Introspection 


The instnjctor allows the student to 
review their actk>ns and decisions by 
presenting them in an appropriate 
manner and alk>wing the student to 
browse through them. This type of 
interventton encourages the student 
to reflect on their problem-solving 
activities. 


Instructor presents 
student/expert 
goal/actton tree to 
student. 


Review, critkiue 


Modeling 


The instructor performs the task; 
modeling for the student the way an 
expert does the activity. An important 
component of modelling is having the 
system arttoulate the decistons it is 
faced with and the strategtos it is 
using to make these decistons. This 
type of interventton is useful in 
making expitoit the strategies the 
expert uses, thereby giving the 
student an example to foltow. 


Instructor 'plays-back' 
experts solutton for 
student. 


Demonstrattons. 

examples, 

counter* 

examples, 

explanation, 

justiftoations 


Coaching 


The instructor breaks in and makes 
suggesttons. When suboptimal 
behavtor or performance is 
recoonized thA "coach" hrpak^ in tn 
give advice. This type of interventton 
is most useful in situattons where one 
wants to give the advtoe designed to 
overcome some specifto weakness 
noted in the student. One ooukf also 
try to discern "what" the nature of the 
weakness is the tutor is responding 
to. 


Instructor provides a 
specifto "hint" or piece 
of advtoe to student. 


Advtoe, hints, 

suggestions, 

choices 


Material Control 


The instructor specifies some 
particular material or problem to 
present to the student. This type of 
interventton is useful for students who 
seem to be having difttouity putting 
basto concepts together to form a 
deeper model of the domain. 


Instructor requests 
some parttouiar 
material or problem be 
presented to the 
student. 


Topto selectton/ 
sequencing, 
task generatton/ 
selectton, 
examples 
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ATT supports acquisition of instructional heuristics by recording an instructor's observati(Mis of a 
student's problem solving activity as well as tfie instructor's intervention specifications. While 
viewing a student's sequence of troubleshooting actions, an instructor can assess the 
appropriateness of each student acticm in terms of its relationship to the expert's goal hierarchies. 
The instructor can also assess the system inquiries and tests initiated by the student as well as the 
student's interpretation of those test results as indicated by changes in their hypothesis space. In 
addition, the instructor can specify an instructional interventicHi following any student action. 

3.2 ATT Environment 

There are four key principles that guide knowledge acquisition in ATT. These principles were 
driven by the knowled^ acquisiticm issues described in Section 2 and are based on current 
research advances in knowledge acquisition, analyses of existing knowledge acquisition and 
instructional systems, and feedback from a numb^ of domain experts and instructors. 

Knowledge acquisition must take place in a simulated environment. In this simulated enviroimient, 
users interact directly with a representaticxi of their domain that emulates their real-worki 
interacticHis and promotes the solicitation of natural and instinctive responses through a direct 
manipulation interface; that is, users take acticms on the objects representing the compcxients of the 
domain, and the dcraain representation's response to their acticm is the same as the user would 
expect in a real draiain. An issue in constructing a simulated environment is the fidelity of the 
computer-based environment to the real-world working environment (Duncan, 1981). 

The approach taken in the ATT prototype is to simulate inforaMtion available in the real-world but 
not necessarily the format in which it is presented. For purposes of tracking problem solving 
performance, component property information is presented only upon request of the user. Since 
information gathering is a critical aspect of diagnostic problem solving, it is important to be able to 
explicitiy represent what information is used and when. This will enable instructional experts to 
better understand student or expert problem solving performance better. ^ 

The knowledge acquisiti<xi tool should employ an action-based solicitation method in which all 
user actions in the simulated envirormient are recorded in sequence and in **pseudo real-time." In 
addition to recwding user actions, the method should also solicit from the user in **pscudo real- 
time" a problem hypothesis space, that is, the set of possible problem causes that the problem 
solver is considering at any given time that is recorded and updated after each user action. 

The knowledge acquisition tool should support knowledge integration by providing instructors 
with multiple, animated views of the knowledge that is completely introspective and can be 
manipulated. The use of a "glass box" architecture (Anderson, 1988; Wenger, 1989) enables users 
to introspect the ccmtent and structure of knowledge. Multiple views allows differeut users to view 
knowledge in a manner appropriate to their context of use. The tool shouki be capable of both 
supporting and integrating multiple input from multiple experts to ensure that the knowledge 
acquired is as ftee from individual biases as possible and appropriate for its specific context of use. 
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One example of accon^lishing this is through the ccmstruction and use of goal-action pioblem 
solving hierarchies. TTie troubleshooting ejqpert can ccmstruct a goal-subgoal hierarchy in relation to 
a sequence of actions just performed to depict his or her troubleshooting strategy. An instructional 
expert can view the same goal-actioa hierarchy to understand a particular expert's problem solving 
approach as the troubleshooting pearformance is animated. The instructional expert can modify the 
goal-action hierarchy fcx instructional purposes or choose to provide the stodent with a view of the 
expat's strategy. The instructional expert can view die student's problem solving perfonnance in 
relation to the expert's performance by accessing a mapping of student actions cmto the expert's 
goal-acti(Mi hierarchy. 

The knowledge acquisition tool should support knowledge construction by building rtjpitjscntations 
of knowledge from observations of user actions. An indirect method of acquiring knowledge ftom 
experts is based on observing experts perform their tasL By observing what actions are performed 
and the consequences of those actions, it is possible for an observer to infer strategic knowledge 
(Gruber, 1991). 

Our instantiation of Gruber's (1991) model of justification constructicm is as follows: 
Justifications for the actions taken by the expert problan solvers arc constructed fiom differences 
in their dynamic problem hypothesis space hoax one action, or set of actions, to another. After a 
user action has been taken, ATT looks at the user's problem hypotiiesis space to see if it has been 
updated as a consequence of that action. If so, ATT then builds a justification for that action as the 
change in the hypotiiesis space. Justifications for die interventions initiated by the instructor during 
a session witii a particular "pseudo student" are constructed from a "snapshot" of the 
configuration of student actions to expert goals and subgoals at the time die intervention was 
initiated, and fix)m observati<ms by die instructor, obtained and recorded in real time, of die 
student's actions taken and their problem hypodiesis space. 
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Section 4 

AIT Knowledge Acquisition Scenarios 



This section contains descripticxis of sample knowledge acquisition sessions tiiat might be 
conducted using the ATT prototype. The scenarios are nanarive-based, witi). figures of the 
windows, boxes, and objects used in die ATT prototype. The purpose of this section is to provide a 
generic story-board that users of the ATI prototype can follow to explore tiie scenarios of the 
various features and metiiods employed in the areas of session startup, problem setup, expert 
problem solving, "pseudo student'' problem solving, and student instmction. 

It is important to keep in mind while reading this section that die purpose of die ATT prototype is 
to facilitate die evaluation of various new and establidied knowledge acquisition metiiods and 
interaction metaphors within the ccmtext of instructional systems development ATT is neither 
exhaustive in scope and coverage, nor are the metiiods and metaphcrs implemented tiie only ones 
possible. Ratiicr, tiie ATT prototype is intended to serve as a tool to help researchers identify 
additional issues and questions ccmceming knowledge acquisition for instractional systems. In 
each of the following knowledge acquisition scenarios, attempts have been made to identify, 
wherever possible, alternate approaches that could be used, or that should be evaluated as 
alternatives, and issues or questions deserving of additional research. The true success of this 
prototype will be measured by tiie help it is able to provide researchers in identifying additional 
important issues and concerns. 



4.1 System Startup 

Working sessions witii die AIT prototype begin witfi tiie specificati(Mi of a problem to develop or 
use. A user can create a new problem ot open an existing problem in tiie "Open Problem'' dialog 
box as depicted in Figures 1 and 2. When the dialog box is initially opened, a process is selected 
by default C*Heat Reactof in Figure 1, the only process example supported in ATT) and presented 
in the "Process" option filed If the user wishes to change the process, clicking <mi the "Process" 
option field presents a pop-up menu listing other defined processes from which tiie user can select 
(see Figure 2). Once a process has been selected, tiie user can eitiier choose an existing problem by 
selecting tiie problem name from tiie "Defined Problems" list box and pressing tiie "Open" 
button, respectively, or else create a new problem by typing the new prd>lem name in the 
"Problem" field and pressing die "Create" buttCMi. 

The first step in creating a new problem is to enter the presenting symptoms in a dialog box as 
depicted in Figure 3. Following input of the presenting symptoms, users are presented with die 
"Process Simulation" window, as illustrated in Figure 4. The "Process Simulation" window 
contains a schematic overview of the process application. Each of the objects in the window is 
selectable, and the content of the infMmation available is dependent upon AIT*s particular 
knowledge acquisition mode. 
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Figure 1. Sesskm Startup 
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Figure 2. Process Selection (not implemented) 
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Figure 3. Probkm Symptom Input Dialog Box 
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Figure 4. Process Simulation Window 



ERIC 



21 

26 



Figure 5 shows a pull-down menu containing a list of possible "Problcm''-rclated activities 
supported in ATT. These options are clustered into activities corresponding to the presenting 
symptoms, the problem setup, expen problem solving, and student problem solving. An alternate 
method of representing these problem-related activities might be to make them omstantly available 
in an icon*based palette. 
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Figure 5. Problem Menu Selections 



4.2 Problem Setup 

As depicted in the "Problem" menu in Figure 5, problem setup consists of two main activities: 
record ("Record Setup") and display ("Display Setup'*) problem setup information. After 
selecting "Record Setup" from the "Problem" menu, Ac user clicks on an object in the simulation 
window (Figure 4) to txing up a proUem setup dialog box as depicted in Figure 6. Note that the 
system mode is indicated in the lower left corner of the message box, located at the bottom of the 
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Figure 6. Probkm Setup Dialog Box 



simulation window. Objects in the simulation window that can be accessed are indicated by the 
higWightmg of that object as the cursor passes over it As can be seen in Figure 6, there are two 
types of setup actions a user can perfonn on a system component: initialize a component property 
value or mduce a fault The type of setup action is selected by the use of tiic mutuaUy exclusive 
radio buttons for "Property" and *Tault," respectively. 

Figures 7 and 8 show the two types of initialization activities supported in ATT's problem setup 
mode. Following selection of a specific property in the "Properties" list box, users can press the 
Edit button, which enables them to edit that property value either by use of a slider for 
continuous values (Figure 7) or by use of a pop-up menu for discrete value options (Figure 9), or 
else tiiey can press "View," which presents them the component property's current value (Figure 
8). The "Momtor" button and option are disabled, as indicated by by its grayed out appearance 
This coding convention of graying out the interface object is used consistentiy throughout AIT to 
mdicate unavailability of the interfiace otgects. 

Figure 9 illustrates how faults are induced in ATT. Users select the component property they wish 
to fault from the "Faults" list box and press the "Edit" button. The user then specifies a value 
using one of tiic two kinds of "Edit Value" interaction objects discussed previously: pop-up 
menus used to present discrete choices (Rgure 9) or a value slider used to input system- 
constrained continuous values (Figure 7). 
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Figure 7. Problem Setup: Edit Component Property 



The other type of problem setup activity supported in ATT is displaying the problem setup 
information for the expert to review (Figure 10), When die user selects **Display Setup" from the 
"Problem" menu, the "Simulation" window is presented, with all selectable objects disabled, 
overlaid with component property values and fault induction values cumendy entered. 



43 Problem Solving 



AIT supports five major acdvides within the context of expert problem solving (see options in 
pull-down menu on Hguie 5). These activities include creating, recording, playing*back and 
deleting expert problem solutions. In addition, an expert can construct and edit a goal-action 
hierarchy for a recorded solution. AFT also supports four major activities within the context of 
student problem solving. Since these problem solving activities are identical to the activities 
involving experts, with the exception that the student docs not ccmstruct a goal-action hierarchy, the 
discussions of the creating, recocxiing, playing-back and deleting problem solutions that follow will 
apply to both students and experts. 

In creating a new problem solution, the user simply enters a new problem solution name in the text 
edit field of the dialog box. This action creates a proUem solution space that needs to be defined 
using the "Record Expert Solution" menu option. 
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Figure 8. Problem Setup: View Component Properly 



When a user begins a problem solving activity, they arc first presented with the window 
combination depicted in Figure 11. This includes the simulation window, now in "Record 
Solution" mode as indicated by the mode message in die lower left comer of die message box, a 
well as an "Action Monitor" window. The message box also contains brief instructions on what 
the user should do next 



The "ActitMi MonitOT" window provides a record keeping environment for the user to review 
during problem solving. This includes a scrollable text box containing the problem symptoms, an 
"Action" panel that presents and supports basic editing cjq)abilities on user actions, and a 
"Hypotiiesis" panel that presents and supports basic editing of problem hypotheses. Action editing 
includes the abUity to view or edit an individual action, attach an explanation to an action, move an 
action up or down within die list of acti<Mis, or delete an action. Each of dwse functions will be 
discussed further later in tiiis section. Hypothesis editing includes specificaticm of the contents of 
the hypothesis display in terms of hypotheses currendy under suspicion, hypotheses currcnUy 
ruled out, or all hypothesis, as well as the ability to select and rule out specific hypotheses. 
Hypothesis editing is nwdal. This means tiiat users have to exit "hypothesis" mode explicitiy 
using die "Done" button before proceeding with other problem solving activities. 
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Figure 9. Probkm Setup: Edit Cratponent Fault 



The first problem solving activity the user engages in is the creation of their initial problem 
hypothesis space. This is accomplished by clicking on an object to bring up the dialog box depicted 
in Figure 12. The user selects a fault hypothesis from the list box, then selects either the "Suspect'' 
or "Rule Out'* buttons, then "Exit." ATT currently uses discrete lists of problem hypotheses. 
However, any type of input could be used* constrained only by what the applicaticHi's device 
knowledge will support 

Figure 13 shows a typical "Action Monitor" window after the expert has created the initial 
problem hypothesis space. After pressing "Done" and exiting "hypothesis" mode, users are 
instructed to click on an object in the simulation window. In "Record Solution" mode, clicking on 
an object in the sunulation window will present them with a problem solution dialog box as 
depicted in Figure 14. 

Figure 14 also shows Uie types of problem solving activities supported in AIT. Users can either 
view, edit or monitor a component property, or else they can view the results of a test on some 
component property. These activities are reflected in Figures 15, 16, 17 and 18, respectively. 
Viewing a property value is a request for the current value of that component's property. Editing a 
component property value is accomplished using the same methods as discussed previously: pop- 
up menus for discrete values and sliders for continuous values. Selecting the "'Monitor^' button 
sends a request to the system to display in continuous and dynamic fashion the property value 
selected in the simulation window. 
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Figure 10. Problem Setup Review 



Selecting the 'Test" option brings up a display of possible tests for that COTiponcnt The user then 
selects the test to be pcrfonned and presses the "View" button and AIT displays the results of that 
test. It is also possible to define new tests in ATT dynamically. One of the choices available in the 
"Options" menu in the title bar is "Define New Test" Selecting this option brings up the dialog 
box depicted in Figure 19. To define a new test the user first selects a component type fiom the 
'Types" list box. Depending upon the component type selected, the "legally" measurable 
properties of that component type are presented in the "Properties" list box, and any existing tests 
arc presented in the "Existing Tests" list box. The user can then select a property to be tested, give 
the new test a name, and press the "Create" button. 

Tnis method of defining a new test is relatively unccmsttained. In all likelihood, this functionality 
would need to be restricted by such things as available tools, practicality, time to test, etc. It was 
included as a demonstration of how this functionality might be implemented: the hierarchical 
chaining of features and properties to produce new entities. 
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Figure 11. Problem Solution Windows 
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Figure 12. Hypothesis Input 



The "Action Monitor" window also continually displays the actions users take in the simulation 
window as well as their updates to the problem hypothesis space. Figure 20 presents an example 
of an "Action Monitor" window following four user actions that have resulted in the ruling-out of 
three of the four problem hypotheses. The user can review each of these actions and either continue 
with the problem solving process by selecting another component or else edit an action in the list 
by selecting it with a single mouse cUck. Once selected, the user can move the action up and down 
in the list (by use of the up and down arrow buttons), delete the action, edit or view the action, or 
else attach an explanation to the action. If the type of action selected is an edit action, the user sees 
the "Edit" button, as depicted in Figure 20, which allows them to change the action's property 
value if they so desire. If the type of action selected is a view, monitor, or test activity, the "Edit" 
button is replaced with a "View" button, signifying that the user cannot change the value 
represented in the action. 
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Figure 13« Probtan Solution Monitor with Hypotheses 
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Figure 15. Problem Solutran Dialog: View Component Property 
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Figure 17. Problem SolutioD Dialog: Monitor Component Property Request 



Figure 21 shows an example of the **Edit Action" dialog box opened when a user selects an 
editable action and presses the "Edit" button. As can be seen, values are edited using the same 
interaction objects discussed previously: pop-up menus for discrete choices and sliders for 
continuous values* 

Figure 22 shows the dialog box presented when the user chooses to view an acticm. In this case, 
the value cannot be edited; only the results of the action are presented 

Figure 23 shows the dialog box presented in response to a request to attach an explanation to a 
specific action. The explanation is entered in fite-form text in a scrolling text entry field 
Explanations are saved along with their associated action for later use in building justifications for 
user actions. 

Another problem-related activity supported in ATT is the playing back of problem solutions. 
Figure 24 contains the primary windows used for review of specific problem solutions: the 
simulation window, a playback control window, and an "Action Monitor" window customized for 
review purposes. 
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Figure 21. Editing an Action 
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Figure 23. Attaching Explanation to Action 



Playback can be executed under user control by using the "Next" button to move to each 
successive action, or else it can be executed under system control Under system control, the user 
can set the playback speed and can still exercise some control over the playback by use of "Start," 
"Pause," and "Resume" buttons superimposed on top of each other in the bottom left of the 
"Playback Control" window. At the beginning of a playback session, the "Playback Control" 
window opens with the "Start" button visible and die "Pause" and "Resume" buttons disabled 
and invisible. Once playback has begun the "Start" buttcwi is replaced by the "Pause" button, 
which allows users to halt the playback ten^xjrarily. Pressing "Pause" causes the "Pause" button 
to be replaced by the "Resume" button, which resumes the playback from the point at which it 
was st<^pcd. The "Reset" buttwi returns the playback to the first action. 
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Dunng the playback process, the first action in the action list is highlighted, as is the object 
representation of the conqwnent refcntd to in the action in the "Simulation" window, ^addition. 
ATT presents the hypothesis set configuration foUowing that action. The ''Explanation" field 
Z'Z ^"fSf"^ "^"^ P^oasly, the explanation field can contain 

^hrZ^l"? • ^ """^f? ^ ^""^ ^"^8 action editing, and (2) differences 

n Ae problem hypothesis space from the previous action (or set of actions) to the present action 
In Figure 24 the action taken was a test of the float device in the surge tank (note that the surge tank 
""1 « smulation wmdow is highlighted; unfortunately the highlighting of the action in the 
acuon field did not reproduce in Figure 24). ITie result of diat test indicated the float device was not 
stuck, so the expert ruled out the hypoAesis that the problem was due to a stuck float 



4.4 Goal Tree Construction and Editing 

The final ATT problem solving activity that experts engage in is the construction and editing of 
their problem solving goal hierarchy. TTus process is iUustrated in Figures 25 through 28. Goal tree 
«roTp^^^^^ f? ^* ^""^^^ ^^'^ represented as an action list in the 
H?^n^ 7 ^^^^ "^^^ ^ ^> ^ ««Phic^ objects (with terse text 

descnptions) m the lower panel of the "Goal Tree" window (the top window in Figure 25). The 

Add Goal or Add Subgoal" button. Pressing those buttons opens a dialog box into which 
experts can entCT goal or subgoal tities (sec Figure 25). Goals and subgoals can also be deleted by 
selcctmg them from tiie list and pressing the "Delete Goal" or "Delete Subgoal" button. Lists of 

It^nn^^^ ^^^.^ ^ ""^"^ ^^^^ ^ selecting. Disjoint 

selections are made by holding down tfie command key during selection. 

S^p'^t"^^^' T """"u"^ ^P*^^^ ^''j"^" dynamicaUy drawn in the upper panel of 
the Goal Tree wmdow as they are created. Figure 26 shows a situation where one malZd three 
subgoals have been created. Once created, the expert then links actions to subgoals and subgoals to 
goals by selecting tl.e entries from two adjacent lists (Le.. goal and subgoal or subgoal and action) 
and pressing the corresponding "Set" button. Figures 27 and 28 contai^ goal trees for which^e 
goal-subgoal. and subgoal-action links have been created, respectively. It is important to note that 
any acdon can be Imked to any number of subgoals. and any subgoal can be linked to any number 

The "Show" buttons are used to highlight links in the "Goal Tree Editor" window as foUows- 

T^^ T ^1^°^^' ^"bfi<«^ ^ edit" "sts. Pressing the contsponding "Show" button 
t'hf ''Shot^LttolT^ " "^^"^ infortnation in the list on the right side of 

The ciment implementation of goal tree constniction was included for exploratory purposes As 
such, the capabUity does have some functional limitations. The goal tree has no functionJu^to 
compress actions or scroU the window. Due to this limitation, only ten actions can be displayed in 
the bottom panel of the "Goal Tree" window regardless of how tLy actions there are in t!^ 
problem solving record. In addition, the goal editor can only handle one expert solution at a time 
Addinonal functionality wiU be required to address these two limitations and enable this capability 
to be evaluated as a vehicle for combining distributed sources of expertise by way of theirgoal 
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Figure 27. Goal Hierarchy Coastractioa: Goal-Sttb|oai Liakias 
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4.5 Student Instruction 



Figure 29 illustrates the possible "Instructor" actions supported in ATT. These activities are 
organized around those activities that support the instructor's understanding of the q)ecific 
problem symptoms, setup and solution, and those related to the instruction of students in specific 
problem solving activities. Reviewing problem symptoms, problem setup, and playing back expert 
problem solutions access the same ATT functionality described in the sccnarici on problem setup 
and solution. The additional functionaKty provided in this menu involves support for the instructor 
in tutorial interactions witii "pseudo students" created in ATT. Pointing to tf»e "Instruct Student" 
option in tbc menu causes a list of "pseudo students" to appear to the right of die arrow in an 
additional selectable list, if students have been created. One merely moves die cursor over one of 
the students and releases tiie mouse button to select one fm: an instructional sessicm. 




Figure 29. Instnictor Mtmu Selectkms 

Once a student has been selected, tiie instructor is presented with die window configuration 

displayed in Figures 30 and 3 1. The window in die upper portion of the figures contains a 
dynamic, graphical display of the relationship between student actions and expcn goals. The cxpen 
goals are displayed in the top panel of die window, while tfje student actions are presented, and 
linked to expert goals if appropriate, in U»e booom panel of die window. In die examples in Figures 
30 and 31, die upper window contains ducc separate goal trees for dealing widi die same problem 
derived from diree different expert solutions. The "Smdent-Insnuctor Session" dialog box in die 
lower portion of Figures 30 and 31 contains the means by which instructors control die 
instructional session, review smdent hypodieses, state their observations, and recommend 
interventions. 

47 

42 

ERIC 



R75- 
Cooling 

ProK>lemi 



Product 

Ftmp LOU 



m 

Reactor 

input 

Probiemi 




opump 
Problems 



TO" 
Product 
Output 
Problem! 



'roduct 

output 

Low 



h/o surge 
rank 
Probiemi 




surge 
rank 

Leuel 
Higi) 



6 Pump 
Problem! 



R75- 
Product 
Output 
Problem! 



RTjrfiirge 

rank 

Problem! 



FIOUJ i! 

tee.e. 



0= 



Mode: 



student Inttruction sii««iAn 



Action: Output of Pipe 6 using Flow is 
100.0. 



Hypotheses 



Surge Tanic is Stucic Float 
Pump is Not running 
Pipe 6 ii Dlutli. e a 



Display 

®flll Goal Trees 

O One Goal Tree 



EKpeil 1 



Hypothesis— 1 
O Correct 
O Incorrect 
<S> Incomplete 
O Missing 
O Other... 



Obserutttinny 



Hction 

O Correct 

® Correct, out of sequence 
O Unrelated to sub goal 
O Unrelated to problem 
O Other... 



Interuenfiont 



flduise/Hint 



Rssist 



Replay Student 



Playback Ewpert 



(prDiiioiJs fl( Uon] | Newt Action 



EMit 



Figure 30. Instructor Istervcatioa WiBdow»— 1 



43 



48 



product 
rtmp Lou. 



STo" 
Cooling 
UJater 
Prot)!emi 



PToouet 
Output 

LOW 



Reactor 
input 
ProDiemi 




/OPump 
Prooiemi 



remporat 

lire ii 



Liuet 11 

875. 



FIOU^ II 



Mode: 



HL 



Student Instn iction Session 



14 



Action: Output of Pipe 6 using Floui Is 
100.0. 



Hypotheses 




Oisplay 

®Ril eooi Trees 

O One Goal Tree 



jEupeil I 



Obserucitions 



interuenttons 



Hypottiesis 1 

0 Correct 
0 Incorrect 
0 incomplete 
® Missing 
0 Otiier... 



r Ret ion 

0 Correct 

® Correct, out of sequence 
O Unrelated to sub goal 
0 Unrelated to problem 
0 Other... 



flduise/Hint 



Rssist 



Replay Student 



Playback Enpert 



Preuious Action 



NeKl R< Uon 



EMit 



Figure 31. Iwlmctor lalervnliM WiMlows~2 



44 

40 



It is topomm to note several features of this fiinctionality tim are explocatoiv in nuure F.r» ^ 

SwB^^ ^ 198«) to grade the iqiresentaiioii of the goal tree. However, we were 

s"^ « .^Ji'ST' 7* •» «^ °^ metaphor for^Sr 

single or multiple goal trees. As such, we decided to enable two difTereituppioaches faf S 

presentat.00. that is. aU trees or a single, selected tree displayed, for STt^^^^lS^ 

comments regaiding the efficacy of the two appioaches So^d, wefoSS^ Sf, 

CmTu^Z^- ™ l-Sdl^^oSand 

rCf:r=-rits^rrr«.Tn^^^^ 



student Instruction Xyf*fnf^ 

fiction: Temperature of Water Source is 
50.0. 



Hypotheses 



Hypothesis 
O Correct 

O Incorre 

O Incompi 

(§) Missing^ 

O Other... 



Display — 

(§) fill Goal Trees 



fiction: Temperature of Water Source is 

50.0. 
Edit flduice: 



Vou should pay closer 
attention to the problem 
symptoms; they will tell 
you where to begin 
looking for the problemj 



OK 



Cancel 



U uiiieiai«ij lu prooiem 
O Other... 



{Tree 



ise/Hint 



ssist 



J Student ] 



Playback Expert 



Pf euious fh Uoi\ 



NeKt fiction 



EKit 



Fifure 32. iMtnKtor latervntioa: Edit Advic* 



45 r 



50 



Given that this is an initial cxploraticMi of these issues, we believe extensive research will be 
required to evaluate not only the efficacy of these approaches but also to help identify the particular 
types of observations that real expert instructors use. In the current implementation, the instructcwr 
can state their observations and reccmmiend a type of intervention after each student action. The 
rationale behind this procedure was to identify a method of cdlecting a large amount of data cm the 
when, why and what of dynamic instruction^ intarventicms. **When'* infonnation is captured in 
tiie specific smdent actions to expert goals configuration. **Why'' information is captured in tiie 
instructor's observaticms about tiie student's iroblem hypothesis space and tiieir last problem 
solving action taken. "What" informaticMi is contained in the type of instruction recommended. 
Theoretically, this data could then be abstracted into general instructional heuristics for a particular 
problem domain. 

The final "Instnictof* activity supported in ATT is "Review Student Session." In this activity, tiie 
instructor is given the opporttmity to review an instructicmal session by providing them with a 
dynamic expert-goal-student-action window, and an "Intervention Summary" window as depicted 
in Figure 33. Unfcatunately, there was not sufficient time to implement the capability to edit a 
student session. 



4.6 Session Shutdown 

The final scenario involves saving the current session, if desired, and shutting down the AIT 
application. ATT has the ability to save all problem setup, expert solutions and goal trees, student 
solutions, and student-instructor interactions attached to the specific problem created or worked on 
to a separate file. At present there is no limit to the number of iroblems that can be created, hence 
there is no limit to the number of problem setups, solutions and interactions as well. To save tiie 
information created or edited during that session, pull down the "Rle" menu and select the "Save" 
option as depicted in Figure 34. To exit ATT once the infonnation has been saved, pull down tiie 
"File** menu and select tiie "Quit" option. If you quit before saving, all information input during 
that session will be lost 



51 

46 



ProQuct 
remp Lou 



rTo 
Cooling 
lUater 
ProDlemt 



>roduct 
Output 




Reactor 
input 
IProbiemt 



OPump 
ProDlemt 



W 
Product 
output 
ProDlemi 



VO surge 




surge 
rank 
Level 
hflh 



remperat 
ureit 

seje. 



Leuei it 
875. 




VOftrge 
rank 
*roblemt 




OPump 
ProDlemt 



Stirce 



B 



Mode: Recor< 



Student-Instructor Session Reuieui 



Hypotheses 



Rction: Leuei of Surge Tonic is 
87.5. 



rObseruotions- 



Kypotheses: Missing 

Rction: Correct, but out of sequence 



Interventions 




Preulous Action 



NeHt Rction 



EHit 



14 



Figure 33. StadeaMBStrBctor Scstkw Review WiMlows 



ERIC 



47 



5; 



Edit Options Problem Instructor 




Figure 34. File Menu Options 



53 



Section 5 
Conclusions 



The current research effort has identified a number of conciusions regarding the requirements of a 
knowledge acquisition tool forinstiucdonal systems. 

First, the training and performance of troubleshooting activities in con5)lex systems appears to 
require the integration of at least three different types of knowledge that will likely come from 
different sources: (1) knowledge about systems and their normal and abnormal behavior ("how it 
works"), (2) knowledge about interacting with these systems to perform a job or task C*how to 
use it"), and (3) knowledge about how to teach students to perform a job or task ("how to teach 

Second, it appears likely that the acquisition of these types of knowledge for use in instructional 
systems can be faciUtated by developing a knowledge acquisition tool that has some or all of the 
following characteristics: 

• Knowledge acquisition shouW take place in a simulated environment in which uscre 
interact directly with a representation of their domain that emulates their real world 
interactions and promotes the solicitation of natural and instinctive responses 
through a direct manipulation interface. 

• The knowledge acquisition tool should employ an action-based solicitation method 
in which all user acticms in the simulated environment as well as their dynamic 
problem hypothesis space are recorded in sequence and in "pscudo real time." 

• The knowledge acquisition tool should support knowledge integration by providing 
users with a glass box architecture with multiple, animated views of the knowledge 
that are completely introspective and can be manipulated. 

• The knowledge acquisition tool shodd support knowledge construction by building 
representations of strategic knowledge, such as justifications, from observations of 
user actions. 

Two examples of justification construction employed in the present system include 1) constructing 
justifications for the actions taken by expert problem solvers from differences in their dynamic 
problem hypothesis space from one action (or set of actions) to another, and 2) constructing 
justifications for the interventions initiated by instructors from a "snapshot" of the configuration of 
student actions to expert goals and subgoals at the time the intervention was initiated, and &om 
observations by the instructor, obtained and recorded in real rime, of the student's actions taken 
and their problem hypothesis space. 
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Section 6 
Recommendations 



The results of the present research effort indicate that there are a number of issues that either 

^^TZ T"^? TZ "^"P^ P"«^"^ ««<^h em« represe^tsl^^^ to 
toT^res^??2«S'H'^^'^^ 

to DC addressed. These additional questions include: How do expert instructor ir^m^i^ « 
student IS in need of soaie type of intervention and what clues^^^S^'^fr^ of 
mtervention to uutiae? What is the scope of the type of intetventio Ju^ by 
How do you dujlay information ftom student and expm problem soS LtiwS^to fSt^ 
recognition by the instructor that an intervention is i^Swhat inf^^rSc^Jp 

unobtrusive way? And most importantly, is it possible, practical, and useful to acereeate 
mstrucuonal expemse ftom specific instructional situati^ekeral i^mu^oS^tics? 

Another issue has to do with justification construction. In the current prototype we have 
m^emented an approach to constructing justifications based on the S^^^Z Gruber 
(1991). However, the question must be raised as to iust how eff«>rive thic -IZTT- ■ 
the true essence of expertise. Is there additional inToi;^l"t^^^ t b^^L^^^^^^ 
justification and are there unobtrusive ways of acquiring this inf<S^ S^e^i^tL 
mteractions with a domain representation? ""«nnanon tnrough natural 

^Zk £! !f ^ ' "^^y *° "^"^^^ ^ I««nt goal trees? Can rhe goal^ 

metaphor be used to faciUtate the integration of distribut^ sources of «^iti^? ^ 

h addition to the global research questions, it is recommended that ftmhcr work be done on the 
ATT prototype m a number of areas. First, the ATT functionality sho^sUgMy e^^L^ to 
aUow for its use m empirical investigations of some of the approacheTand Zf^lT^^ 
»nted litis wiU involve impLenting the abdity toST^c^nt a^^^of 
accomplishing the same objective selectively (c.g., disptay eitha aulcS^T^Jfa J J , ^ 

k^wledge as weU .s d« implcmcnutio, of „y „w devel^X^. 
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Appendix 
Instructions for Software Maintenance 



A.l Software Organization 

AU files related to this project arc kept in a folder named "ATT Folder." The contents of this folder 
arc shown in Figurc A-1. The ATT-INrr file contains some basic functions for loading and saving, 
and definitions of the appropriate paths for all files. The DEFS YSTEM folder contains the code 
for defining and maintaining systems. All the source files of this projea are included in a system 
called the ATT system. The details of using the system feature are described below. The SOURCE 
folder contains the source files and the BIN folder contains the compiled vereions of the project 
files. *^ 




Sourc* 



Bin 4it-init.fw1 




d*fsyst*m ait-init.lisp 



Figure A-1. The Coateats of the AIT FoMer 



A.2 Maintaining the Source Code 

The source files are maintained using a public domain DEFSYSTEM software modified to 
enhance its ease of use by Honeywell SSDC. The Ust of files that constitutes the system and the 
order in which they should be loaded are described in the file SYSDEFS. The SYSDEFS file is ir 
turn maintained by the developer through the Edit System dialog box and the Systems menu. The 
Edit Systems dialog box (shown in Figure A-2) shows a listing of all files that make up the 
system. By cUcking on a "Rx^dio" button, the developer can switch between displaying the files in 
alphabetical order or in the order in which they arc to be kwlcd. 
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Edit <fll1> 
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® Edit Files [ Load Sgs " 
O Load Files 
O Compile Files 



Compile Sys 



O Hardcopy Files ( Do it "] 



Figure A-2. The Edit Systca Dialof Box 



Other options avaiUblc in the dialog box allow the developer to load, edit, compile and hanicopy 
one or more files. These operations arc pcrfonncd by first selecting the files, setting the appropriate 
"Radio" button and finally clicking the "Do It" button. Qicldng on the "Load Sys" button at any 
time loads all the files that have changed since the system was last loaded. Clicking on the 
"Compile Sys" bunon compiles all source files that have changed since the last compile. 
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Sgst(»ms 



Edit System 



Rdd File 
Delete File 



Saue System Def 
Load System Def 

Figure A-3. The S^ou Mcbu 

The menu items under the "Systems" menu allow the developer to bring up the "Edit System" 
dialog box. New ffles can be added by choosing the "Add Files" option. A file dialog box comes 
up allowing the new file to be selected. The new file is inserted just above the first selected item in 
the "Edit System" dialog box. If no file is selected in the "Edit System" dialog box, then the new 
file is added at the end of the list New files cannot be added to the system when the "Edit 
System" dialog box is displaying a sorted list of files. Files may be deleted from the system by 
selecting them and choosing "Delete File" from the "Systems" menu. 

Adding and deleting files from the system does not by itself change the SYSDEFS file. After 
making several additions/deletions, Ae SYSDEFS file can be updated using the "Save System 
Def item from the "Systems" menu. Also, if the systems file has been updated manually, then 
that information can be loaded using the "Load System Def option under the "Systems" menu. 



A.3 Loading of Code into LISP Image 

The AIT software was developed under Macintosh Common USP Version lObl with patches 1 - 
3. AU indications are that MCL V2.0 will be released in its final fonn in early 1992. We do not 
anticipate any compatibility iroblems witfi the released version. 

When LISP is launched, it automatically loads the INTT file from its home folder. A sample INTT 
file is included in our software package. Inside the INTT file are commands to load another file 
AIT-INIT (described earlier). This in tum k>ads all the files necessary for ATT to run. 

When loading is complete, the welcome prompt is dispUycd in the LISP Ustener. At this time the 
current package will be the ATT package. A fiinction caUed STARTUP has been defined in the file 
STARTUP.LISP. STARTUP creates all the necessary windows, sets up the menubar and kicks 
off the simulation. STARTUP also looks for and loads a file HEAT-REACTOR from the home 
directory. This file contains all the problems, solutions and goal trees. Anocher file that STARTUP 
looks for is REaRCPICT. This is a PICT format file containing the picture of the heat reactor. 
This file can be modified using any drawing appUcation such as MacDraw. If any changes are 
made to REaRCPICT, care must be taken to save it back as a PICT file. 
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A.4 Making an AITImage 



An image file is an application that contains all the functionality of the LISP environment as well 
as any other code that was loaded befcxe making the image. Its advantage is that it reduces the tinie 
it takes to load user defined code. An image containing the ATT code can be easily made by 
invoking the function MAKE- AIT-IMAGE. You will be pron^ted for an inaage name. 
Subsequent changes to ATT can now be made by invoking this image. 

The image is setup to load the INTT file automatically on being launched This has the effect of 
loading all changes made to the source code since the last image file was created. The STARTUP 
fiincticm is also called autcxnatically to create all the windows and set up the menubar. 

A.5 Making the AIT Application 

A LISP application is defined as one that contains all the LISP functions except for the compiler, 
inspector and other such developer features. An application is the form in which software is 
delivered. Users of applications do not require USP or a license to use it The ATT application can 
be made by calUng the function MAKE-ATT-APP after loading all the ATT code. This function can 
also be called in a previously made image. 

When the application s launched, the STARTUP function is called to setup all the windows and the 
menubar automadcally. The AIT application should be distributed along with the file 
RECIRCJ^ICT. If example problems are to be included, the file HEAT-REACTOR must also be 
included. 
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